DOCDHEKT HESOHE 



KD 130 188 



CG 010 867 



AOTHOR 
TITLE 



PUB DATE 
NOTE 

EDRS PRICE 
DESCRIPTORS 



Piercy, Fred P. 

Relationships Among Counselor Effectiveness 

Self-Hatings^ Peer Ratings^ Supervisor Ratings, and 

Client Ratings. 

[74] 

1Up. 

MF-$0.83 HC-$1.67 Plus Postage. 

♦Behavior Rating Scales; *Counseling Effectiveness; 
♦Counselor Evaluation; Counselor Performance; 
Counselor Training; Evaluation Methods; Peer Groups; 
Research Projects; *Statistical Bias; Supervisors 



ABSTRACT 

The purpose of this study was to investigate the 
comparibility of counselor effectiveness ratings made by four 
different groups. This study examined the relationships among student 
counselors" self-ratings, peer ratings, supervisor ratings, and 
client ratings on the Counselor Effectiveness Scale, Form 2, a 
semantic differential rating scale appropriate for immediate use with 
raters of varying sophistication. No significant relationships were 
found between pairs of rating groups on the total score of the CES. 
In addition, no significant difference was found among the mean 
ratings made by the four rating groups. Implications for counselor 
effectiveness research and counslor training are discussed. 
(Author) 
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Abstract 



The purpose of this study was to investigate the compar- 
ibility of counselor effectiveness ratings made by four dif- 
ferent groups. This study examined the relationships among 
student counselors^ self -rratings , peer ratings, supervisor 
ratings, and client ratings on the Counselor Effectiveness 
Scale, Form 2, a semantic differential rating scale appro- 
priate for immediate use with raters of varying sophistication. 
No significant relationships were found between pairs of rating 
groups on the total score of the CES . In addition, no signi- 
ficant difference was found among the mean ratings made by the 
four rating groups. Implications for counselor effectiveness 
research and counselor training are discussed. 



Relationships Among Counselor Effectiveness 
Self-Ratings, Peer Ratings, Supervisor Ratings, 
and Client Ratings 
While ratings are the most frequently used of all the cri- 
terion measures in counselor effectiveness research, the 
identity of those doing the rating varies widely. Groups 
most commonly rating counselor effectiveness in previous 
research include counselors, peers, supervisors, and cli- 
ents. Many researchers contend that each of these groups 
can provide valid ratings of counselor effectiveness. How- 
ever, others suggest that different groups of people see 
counseling differently, and therefore may rate counselor 
effectiveness differently. Factors hypothesized to relate 
to differential ratings include differing definitions of 
counseling (Knupfer, Jackson, and Krieger, 1959) , pri- 
orities (Johnston, 1966) , outside cues (Kiesler, 1966) , 
internal references or anchors (Klein and Cleary, 1967), 
biases (Patterson, 1967) , dependent judgments (Chinsky 
and Rappaport, 1970), and theoretical orientations (Ward, 
1974). 

If different groups of raters do vary greatly on 
their ratings of counselor effectiveness, it may be that 
effectiveness is in the "eye of the beholder." If this 
is the case, results of studies using effectiveness ratings 



of differing groups may not be comparable. In other words, 
the results of studies on counselor effectiveness could be 
affected by the specific group doing the rating. Con- 
ceivably, researchers could be inadvertently biasing their 
results by employing one group of raters instead of another.. 

While some data support the use of ratings made by 
peers, supervisors, clients and counselors themselves, con- 
flicting data also exist. In addition, it is unclear to 
what extent these different rating groups agree on ratings 
of counselor effectiveness. Information is contradictory, 
and often represents the piecemeal byproducts of more ex- 
tens ive investigations . 

Carkhuff has stated that "all science begins with the 
sensory experience of the scientist. Rather than take it 
for granted, the perception of the perceiyer must be 
studied" (Ivey, 1971, p. viii) . The present study was an 
attempt to examine such perceptions by systematically in- 
vestigating the relationships among counselor effectiveness 
self-ratings, peer ratings, supervisor ratings, and client 
ratings. 
METHOD 

Instrumentation 

The Counselor Effectiveness Scale (CES) developed by Ivey 
(1971) was the counselor effectiveness rating instrument 
used in the present study. It is a semantic differential 
instrument containing 25 items, with each item having 
seven steps between a positive counselor trait (e.g.. 



helpful) and a negative counselor trait (e.g., unhelpful). 
The CES, available in parallel forms, provides a global 
measure of counselor effectiveness and is appropriate for 
immediate ' use with raters of varying sophistocation. 

An estimate of the validity of the CES was initially, 
based on the extent to which naive raters (N=18) could 
differentiate between a videotape of a rationally defined 
"good" counseling model and a videotape of a rationally 
defined "poor " counseling model. On both forms of the CES . 
the raters were able to differentiate between the video- 
taped models beyond the .001 level of significance. Form 2 
was used in the present study because of its slightly greater 
range of differentiation. 
Subjects and their Selection 

The rating groups employed in the present study were Univer- 
sity of Florida student counselors, their peers, their 
supervisors, and their clients. These groups. and their 
selection are ^described below: 

1. Student Counselors — The counselor self-rating 
group was composed of 27 graduate students in the Depart- 
ment of Counselor Education at the University of Florida. 
At the time of the study these student counselors were in- 
volved in a practical counseling experience, either a prac- 
ticum or an internship, through the Department of Counselor 
Education. Graduate student counselors working in elementary 
schools were not included in this study because of the 
difficulty in securing client ratings from elementary school 
children* a 



V,. The number of student counselors that each supervisor 
sees in individual supervision varies widely. In this 
study, one counselor was randomly selected from each of 27 
supervisor's pool of supervisees. This stratified random 
sampling of student counselors was employed in order to 
obtain independency of measurements, as this procedure 
makes certain that no two counselors are rated by the same 
supervisor. 

2. Peers — The 27 peers in this study were also 
graduate students at the University of Florida. Like the 
counselor subjects, the peers were involved in either a 
practicum or internship. 

Peers were randomly selected from those individuals 
attending each particular counselor subject's group super- 
vision session. In some cases, counselor education grad- 
uate student counselors attend agency staff meetings in- 
stead of departmental group supervision. In such cases, 
the corresponding peer counselor was randomly selected 
from those students in the counselor subject's work, setting . 
All peers were either in group supervision or in a work 
setting with the student counselors for a minimum of four 
weeks. 

3. Supervisors — The 27 supervisors in the present 
study were all from the Department of Counselor Educa- 
tion at the University of Florida or from the student 
counselor's work setting. All had been individual super- 
visors to the student counselor subjects for at least four 
weeks. 7 



4. Clients — The clients in this study were indi- 
viduals whom the counselor subjects saw in counseling for 
a predetermined week. Clients must have been seen by a 
counselor subject for a minimum of two sessions to be 
included in the study. One randomly selected client 
rating for each student counselor was used in the data 
analyses. 
Design 

The 27 supervisors each rated a randomly selected counselor 
education graduate student supervisee on the Counselor 
Effectiveness Scale. Likewise^ these 27 student counselor 
supervisees were rated by themselves/ 27 randomly selected 
peer student counselors/ and 27 clients. The relation- 
ships among groups on the total score of the CES was 
ascertained by use of Pearson product-moment correlations. 
A one-way analysis of variance was employed to test whether 
the mean ratings for each rating group differed signifi- 
cantly from those of the other rating groups. 
RESULTS 

The correlations between the CES ratings for each possible 
pair of rating groups is included in the inter correla- 
tion matrix presented in Table '1. As may be seen in Table 
1/ no correlation exceeded .37/ the critical value for 
significance at the .05 level. ThuS/ no significant 
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relationships were found among the overall counselor effec- 
tiveness ratings made by counselors themselves, peers, 
supervisors, and clients. Similarly, when a one-way 
analysis of variance was performed on the mean CES ratings 
for each group, no significant differences (F=2.33, 3,104) 
were found. In other words, no significant relationships 
were found betwe^en pairs of rating groups in terms of 
rating agreement, nor were the mean overall student coun- 
selor ratings made by each group found to be significantly 
different. 
DISCUSSION 

Methodological Considerations 

\niy would the results of this study be at variance with 
other studies finding significant correla:tions between 
rating groups? The rating instrument used in the present 
study may be one reason. For example, it may be more 
difficult to. obtain intra-group agreement on global 
effectiveness ratings than on ratings of specific beha- 
viors or taped responses. A global rating scale such as 
the Counselor Effectiveness Scale used in the present 
study seems likely to allow more room for the rater's 
own subjective preferences to color his ratings than would 
a scale focusing on specific behaviors. It is also possible 
that insufficient contact was established between the 
counselor and the other raters to expect high agreement. 

While such methodological weaknesses should be con- 
sidered in discussing the results of the present study. 



one strength should also be mentioned. In this study, no 
rater rated more than one counselor subject, and no two 
individuals from the same rating group rated the same 
counselor subject. The resulting independency of measure- 
ments appears to strengthen this study. However, inde- 
pendency of measurement procedures are not always found 
in studies reporting significant correlations among rating 
groups. It is not known to what extent dependency of 
measurements, a statistically unsound procedure, might 
have resulted in spuriously high correlations among rating 
groups in other studies. 
Conclusions and Implications 

The major finding in the present study was that counselors ^ 
peers, supervisors, and clients did not agree on overall 
ratings of counselor effectiveness as measured by the 
Counselor Effectiveness Scale. While this finding may be 
an indictment against the global rating instrument or the 
methodology employed, other explanations should be explored. 
The results seem to support the view that effectiveness 
may indeed be in the "eye of the beholder." It may be, 
then, that different groups having differing definitions 
of "effective counseling" growing out of individual 
orientations , biases , internal anchors , or other differ- 
ences. On the other hand, it simply may be that a coun- 
selor's effectiveness varies greatly from time to time 
and from client to client/ making it difficult for different 
groups to agree on that counselor's "general" effectiveness. 

Er|c 10 



Whatever the reasons, the results of this study con- 
tradict the general assumption that the counselor effec- 
tiveness ratings made by counselors, peer?, supervisors, 
and clients are comparable. The results clearly indi- 
cate that there were no significant i elationships among 
the CES ratings made by these groups. The current prac- 
tice of comparing counselor effectiveness studies which 
use different rating groups does not appear justified. 
Since ratings made by different groups may not be con- 
sistent with one another, studies using one particular 
rating group may only be comparable to other studies 
using the same category of raters. 

It also appears unwise to assume that a counselor 
effectiveness study employing one group of raters will 
necessarily result in the same findings if a different 
rating group is substituted. If the results of studies 
vary as a function of the rating group employed, it is 

/J 

critical for researchers to take into account the group 
doing the rating. 

These results have implications for counselor train- 
ing. If three groups of counseling-oriented individuals 
(supervisors, student counselors, and peers) cannot agree 
on who are effective counselors, what direction should 
counselor training take? Is it even possible to train 
people to be effective counselors when "effective counsel- 
ing" jnay mean many things to many people? A number of 
researchers have studied and articulated various facilitative 



conditions of effective counselors. However, student 
counselors and counselor educators should be aware of the 
extent to which they themselves agree on the degree to 
which such characteristics are manifested by counselors- 
in-training. Agreement as to what is the best theoretical 
framework or specific technique is neither possible nor 
necessarily desirable. But certainly agreement among pro- 
fessionals regarding the desired end productf "effective 
counseling," would seem critical if a counselor training 
program is to move in a positive direction. 
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TABLE 1 

Intercorrelation Matrix on CounvSelor Effectiveness Scale, 
Form 2, Total Score 



Student 

Counselors Peers Supervisors Clients 

.20 .08 .32 

•26 .23 
.12 



Student Counselors 
Peers 

Supervisors 
Clients 
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